This invention relates generally to intermediate nodes of computer networks and, more specifically, to maintaining order among requests issued over an external bus of an intermediate node of a computer network.
A computer network is a geographically distributed collection of interconnected subnetworks for transporting data between nodes, such as computers. A local area network (LAN) is an example of such a subnetwork; a plurality of LANs may be further interconnected by an intermediate network node, such as a router or switch, to extend the effective xe2x80x9csizexe2x80x9d of the computer network and increase the number of communicating nodes. The nodes typically communicate by exchanging discrete frames or packets of data according to predefined protocols. In this context, a protocol consists of a set of rules defining how the nodes interact with each other.
Each node typically comprises a number of basic systems including a processor, a main memory and an input/output (I/O) system. Data is transferred between the main memory, processor and I/O system over a system bus, while data transactions within the I/O system occur over an external bus, such as an I/O bus. Each bus typically consists of either address, data and control lines, with the control lines carrying control signals specifying the direction and type of transfer, or a pair of unidirectional signals for passing bus packets containing address, data and control such as in the case of HyperTransport (HPT) bus. For example, the processor (i.e., a source) may issue a read transaction to request the transfer of data from an addressed location on an I/O device (i.e., a target) coupled to the I/O bus and over the system bus to the processor. The processor then processes the retrieved data in accordance with instructions that may have been obtained from main memory. The processor may thereafter issue a write transaction requesting that the results be stored in, e.g., an addressed location in the main memory.
Some buses operate in an xe2x80x9catomicxe2x80x9d manner such that the source is granted exclusive access (i.e., control) to the bus until the transaction is complete. However, an atomic bus may potentially waste bus cycles, particularly when waiting for data in response to, e.g., a read request. In a split transaction bus, on the other hand, the source relinquishes control over the bus once the request is sent and an independent response to the request is subsequently returned to the source. Here, the target acquires control of the bus to return the response to the source. The split transaction bus thus essentially enables a transaction to be divided into at least two transfers: the request and the response.
Devices coupled to the split transaction bus typically include common sets of resources (such as buffers or queues) used to store request and response transfers sent over the bus. It is possible that some of the resources may be consumed by these transfers, thereby causing a deadlock situation on the bus. To obviate such a situation, transactions sent over the bus may be reordered. However, re-ordering of transactions over a split transaction bus may result in inconsistent data accesses that, in turn, may adversely impact performance of the system and node.
For example, a transaction re-ordering situation that may result in inconsistent data accesses is a read transaction that requests xe2x80x9cpre-fetchingxe2x80x9d of the contents of a block of address locations. If a write transaction is allowed to xe2x80x9cpassxe2x80x9d the read request, then the actual read access will retrieve data having a value indicative of the results of the write request. Another situation where it is necessary to maintain the order of read requests to avoid inconsistent data accesses involves requests directed to a similar address, e.g., the xe2x80x9cheadxe2x80x9d of a first-in, first-out (FIFO) queue. The data xe2x80x9cwordxe2x80x9d present at the head of the FIFO changes between an initial read request and a subsequent read request Accordingly, if those requests are allowed to get xe2x80x9cout of orderxe2x80x9d, the result of the initial read request would be a subsequent data word rather than the intended initial data word.
In the case of a split transaction bus, such as the conventional peripheral computer interconnect bus or HyperTransport (HPT) bus, it is possible for transactions to be reordered due to the use of virtual channels and ordering rules defined for the bus. In particular, it is possible for a read request followed by a write request to the same address to appear xe2x80x9cout of orderxe2x80x9d at their destination. For example, assume a target on the HPT bus includes a control status register (CSR) that contains status information for use by a source on the bus. Assume also that the source stores (via a write request) a particular value in the CSR and then subsequently retrieves (via a read request) the content of that CSR location to determine if it is appropriate to issue another transaction. An example of a typical instruction code sequence for this application is:
Write MACREG1  less than value greater than 
Read MACREG1
If(bit 3=j) then . . .
Write MACREG1  less than new value greater than 
The first write instruction is directed to writing a register (MACREG1) with a particular value. The next instruction is directed to reading the value of that register to check status; e.g., if (bit 3=j) then, etc. Thereafter, a subsequent write instruction is executed to write a new value to the MACREG1 register. Because of the ordering rules on the HPT bus, it is indeterminate as to which operation will occur first. Specifically, the HPT ordering rules allow write requests (e.g., posted requests) to xe2x80x9cpassxe2x80x9d any other request (e.g., non-posted requests) thereby allowing the transactions to be reordered at a device (such as a repeater) on the HPT bus to avoid deadlock. Thus, it is possible for the subsequent write request to pass the pending read request. Clearly, this may be problematic.
A typical solution to this problem is to require completion of the read request before the write request is issued. That is, to ensure ordering between transactions that may be reordered, software (such as a HPT device driver executing on a processor) is configured to defer (xe2x80x9chold offxe2x80x9d) issuing the subsequent write request until the pending read request completes. However, this approach is inefficient and degrades performance of the system since the processor is likely to stall program execution until the read request completes. The present invention is directed to a technique that ensures ordering of transactions without adversely impacting system performance.
The present invention relates to an apparatus and technique for off-loading responsibility for maintaining order among requests issued over a split transaction bus from a processor to a split transaction bus controller of an intermediate network node or any general purpose processor subsystem, thereby increasing the performance of the processor. To that end, the present invention comprises an ordering circuit that enables the controller to defer issuing a subsequent (write) request directed to an address on the bus until all pending (read) requests complete. By off-loading responsibility for maintaining order among requests from the processor to the controller, the invention enhances performance of the processor since the processor may proceed with program execution without having to stall to ensure such ordering. The ordering circuit maintains the order of the requests in an efficient manner that is transparent to the processor.
Specifically, the ordering circuit checks each write request that is issued by the processor to the split transaction bus to determine if there is an outstanding read request directed to the same address currently in progress (i.e., outstanding) over that bus. The circuit includes a data structure (i.e., table) configured to track outstanding requests issued over the split transaction bus. If it is determined that the write request is directed to an address of any outstanding read request, a conflict detection circuit asserts a conflict bit. The write request and the conflict bit are loaded into a first-in first-out (FIFO) buffer adapted to store commands to be transmitted over the split transaction bus.
The ordering circuit further includes a register having a plurality of bits, wherein each bit corresponds to a pending read request sent over the split transaction bus. According to the invention, assertion of a bit of the register and the conflict bit causes the write request to wait until all pending read requests complete before being sent over the split transaction bus. If additional read requests are issued to the same address, they must wait until the write request has been sent to the bus before they can be sent. In this case, the additional read requests may be loaded into the same FIFO as the write request.
In a situation involving read requests to different addresses, the requests can be sent over the split transaction bus before the write request, but the write request must wait for all pending read requests to complete before it is allowed to be sent. This condition is restrictive in the sense that write requests may wait an extended amount of time before being sent over the split transaction bus, which is illustratively an external input/output (I/O) bus. Thus, the present invention ensures proper ordering over an external split transaction I/O bus, such as the conventional peripheral computer interconnect bus or the HPT bus.
Advantageously, the novel ordering circuit provides a hardware solution that off-loads the requirement of a processor to maintain order among potentially conflicting requests issued to I/O devices coupled to an external I/O bus. For example, the processor may issue a pre-fetch read request to an address and forward that request to an I/O controller. The processor may then issue a write request to that address and not have to endure any latency (i.e., the processor may continue processing other information) since the controller assumes responsibility for ordering of those requests. The inventive circuit also reduces the complexity needed by the controller to ensure ordering of transactions, while facilitating increased performance over the bus.